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1.  INTRODUCTION 

The  NIST  Algorithm  Testing  System  (ATS)  is  a software  package  for  evaluating  the  performance 
of  geometric  fitting  software.  The  ATS  provides  basic  capabilities  for  comparing  the  results  of 
different  fitting  routines  to  a set  of  reference  algorithms.  The  ATS  allows  the  user  to  generate 
test  data  for  various  geometries  with  simulated  form  errors  and  simulated  measurement  errors; 
run  test  data  through  the  reference  algorithms;  import  fits  computed  by  other  algorithms;  and 
compare  the  results  of  algorithms.  The  system  was  developed  to  run  on  PC  compatible 
computers.  This  document  applies  to  Version  1.1  of  the  ATS  system  and  describes  how  to  install 
and  run  the  ATS  on  a PC  compatible  computer. 

The  ATS  is  available  from  NIST  on  a formatted  floppy  disk.  This  distribution  disk  contains 
Version  1.1  of  the  Algorithm  Testing  System.  The  disk  includes  compressed  files  of  the  ATS 
executable,  ATS  source  code,  this  User  Guide  and  several  other  documents,  and  data  files. 

The  ATS  was  developed  as  part  of  NIST’s  contribution  to  an  ongoing  effort  by  industry  standards 
groups  to  establish  a standard  for  testing  software  used  in  inspection  systems.  The  project  is 
sponsored  by  the  Navy  Manufacturing  Technology  Program  and  by  NIST  internal  funding.  In 
the  short  term,  the  project  goal  is  to  develop  a production  version  of  the  Algorithm  Testing 
System.  The  draft  repon.  Requirements  for  a Production  Algorithm  Testing  System,  provides  a 
description  of  the  production  version  of  the  ATS.  The  long-term  goal  is  to  develop  a standard 
set  of  reference  algorithms  that  can  be  used  for  testing  software.  This  achievement  would  allow 
industry  to  have  a standard  method  for  testing  software. 


1.1  System  Requirements 

In  order  to  run  the  ATS,  you  must  have  an  IBM  PC  compatible  computer.  The  minimum 
configuration  necessary  is  as  follows: 

80286  or  higher  cpu 

DOS  Version  3.3  or  later 

Hard  disk  with  at  least  2 MB  free  space 

EGA  or  VGA  graphics 

High  density  5 1/4"  floppy  drive 

512  KB  free  memory  (recommend  minimum  of  640K  installed  memory) 

Optional  equipment: 

80  X 87  Math  co-processor 
Mouse 
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A math  co-prcx;essor  is  not  required.  However,  the  ATS  will  use  one  if  it  is  present.  A mouse 
is  not  required  but  is  recommended.  To  install  the  software  as  it  is  distributed,  you  must  have 
a high  density  5 1/4"  floppy  drive  (other  formats  are  available  upon  request). 


1.2  Features 

The  Algorithm  Testing  System  features  the  following  implementations, 
o Least-squares  fitting  algorithms 

o Data  generation  with  simulated  form  errors,  measurement  strategy,  and 
measurement  error  models 
o \\^ndow-based  user  interface  within  the  ATS 
o Data  management  capabilities,  including  data  import  and  export 
o Residual  analysis  of  a fit 
o Detailed  comparison  of  fit  results 


1.3  Overview 

Section  2 of  this  document  describes  the  procedure  for  installing  the  Algorithm  Testing  System 
from  the  distribution  disk  onto  your  system.  Section  3 contains  a brief  overview  for  starting  and 
operating  the  system.  Section  4 contains  a detailed  description  of  the  system  operation.  There 
are  four  appendices  at  the  end  of  the  document.  Appendix  A contains  a description  of  the  ATS 
file  formats.  Appendix  B contains  a description  of  the  ATS  file  names.  Appendix  C contains 
details  on  the  parameters  of  the  data  generator.  Finally,  Appendix  D describes  the  parameters 
reported  by  the  fit  result  analysis  and  comparison. 


2.  INSTALLATION 

The  installation  diskette  contains  multiple  compressed  files  and  is  distributed  on  an  IBM- 
formatted  disk.  The  distribution  diskette  includes  the  NIST  Algorithm  Testing  System;  this  User 
Guide-,  the  document,  Form  Error  Models  of  the  NIST  Algorithm  Testing  System^-,  the  draft 
report,  Least-Squares  Fitting  Algorithms  of  the  NIST  Algorithm  Testing  System;  and  various  data 
files.  The  documents  included  are  in  a WordPerfect  5.1  format. 


‘Algeo,  M.  E.  A.  and  Hopp,  T.  H.,  Form  Error  Models  of  the  NIST  Algorithm  Testing 
System,  NISTIR  4740,  National  Institute  of  Standards  and  Technology,  Gaithersburg,  MD, 
January  1992. 
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The  ATS  is  distributed  in  a compressed  format  and  must  be  expanded  to  be  used.  The 
INSTALL  script  on  the  distribution  diskette  simplifies  this  process.  The  INSTALL  script  can 
be  used  with  one  or  two  arguments,  as  follows: 

Method  1:  Using  INSTALL  with  one  argument 

Change  to  the  floppy  drive  that  contains  the  ATS  distribution  diskette  and  enter 
INSTALL  [target  directory]. 

For  example,  if  the  distribution  diskette  is  in  drive  A:  and  you  want  the  ATS  installed  in 
C:\ATS,  then  enter  the  following  commands: 

A:<ENTER> 

INSTALL  C:\ATS<ENTER> 

Method  2:  Using  INSTALL  with  two  arguments 

From  any  drive,  specify  the  absolute  path  to  INSTALL  and  enter  [source 
drive] iINSTALL  [source  drive]  [target  directory]. 

For  example,  if  the  distribution  diskette  is  in  drive  A:  and  you  want  the  ATS  installed  in 
C:\ATS,  then  enter  the  following  commands: 

AiINSTALL  A:  C:\ATS<ENTER> 


3.  OPERATION 

This  section  of  the  document  contains  an  overview  for  starting  and  operating  the  Algorithm 
Testing  System.  For  more  detailed  instructions  on  using  the  ATS  refer  to  Section  4. 


3.1  Running  the  ATS 

The  command  to  run  the  ATS  is  ats.  For  example: 

C:\ATS\ATS<ENrER> 

starts  the  ATS  executable  in  the  C:\ATS  directory.  The  ATS  starts  up  with  a menu  bar  at  the 
top  of  the  screen  displaying  five  menu  items.  The  menu  bar  is  your  primary  access  to  all  the 
menu  items  and  menu  commands.  Each  menu  item  has  a pull-down  menu  associated  with  it. 
The  pull-down  menus  contain  commands.  In  this  document,  all  ATS  commands  will  be 
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identified  by  the  menu  item  name  and  command  name,  separated  by  a vertical  bar.  For  example, 
the  command  to  exit  the  ATS  is  denoted  programIquit. 

Here  is  how  you  choose  menu  commands  using  the  keyboard: 

1 Press  <F‘10>.  This  function  key  makes  the  menu  bar  active. 

2 Use  the  <leftIright>  arrow  keys  to  select  the  menu  item  you  want  and  then  press 
<ENTER>  to  obtain  the  pull-down  menu  with  the  commands.  The  ^OWN  ARROW>  can 
also  be  used  to  pull  down  command  menus. 

As  a shortcut  for  this  step,  you  can  just  press  the  highlighted  letter  of  the  menu  item.  For 
example,  from  the  menu  bar,  press  T to  move  to  and  display  the  TEST  menu. 

3 Use  the  <downIup>  arrow  keys  to  select  the  command  you  want.  Then  press  <enter>. 

Again,  as  a shortcut,  you  can  just  press  the  highlighted  letter  of  a command  to  choose  it 
once  the  menu  is  displayed.  At  this  point  the  ATS  either  carries  out  a command,  displays 
a dialogue  box,  or  displays  another  menu. 

4 To  cancel  an  action,  press  <ESO. 

5 To  close  a menu,  press  <ESO. 

You  can  also  use  a mouse  to  choose  menu  items  and  menu  commands. 

1 Click  on  the  desired  menu  item  to  display  the  menu  commands. 

2 Click  on  the  desired  command. 

3 To  cancel  an  action,  press  <ESC>. 

4 To  close  a menu,  press  <ESO. 

Most  commands  result  in  a window  being  displayed  by  the  ATS.  The  four  main  window  types 
are  test  descriptions,  data  sets,  fits,  and  analysis  results.  All  interaction  with  the  ATS  is  through 
these  windows.  Each  window  type  has  its  own  particular  procedures,  but  several  features  are 
common  to  all  main  window  types. 

These  common  window  procedures  are  as  follows: 

1 Any  selected  window  can  be  removed  from  the  display  by  pressing  CESO  and  selecting 
<OKAY>. 

2 Any  window  can  dragged  or  moved  using  the  mouse.  Select  the  border  of  the  window 
with  the  mouse  and  move  the  window  while  holding  down  the  mouse  button. 

3 Any  window  can  be  selected  with  the  mouse.  A window  is  active  when  its  border  is 
highlighted.  Move  the  mouse  to  the  window  until  highlighted.  Only  one  window  may 
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be  active  at  any  time.  Clicking  on  the  border  of  an  active  window  in  the  background  will 
bring  it  to  the  foreground. 

4 The  function  key  <F-5>  sequences  through  displayed  windows. 

3.2  Basic  Procedures  for  Using  the  ATS 

The  capabilities  of  the  ATS  are:  defining  a test  description  for  a given  geometry,  generating  a 
data  set  for  the  test,  fitting  the  test  data  set  using  the  least  squares  method,  and  analyzing  the  fit 
results.  Usually  you  will  use  the  ATS  by  going  through  each  of  the  above  steps  in  sequence. 

You  define  a test  description  by  selecting  the  TESTInew  command.  A test  description  window 
is  displayed.  This  window  allows  you  to  select  a geometry  type.  When  you  select  a geometry 
type,  you  also  must  enter  numeric  parameters  defining  the  ideal  (nominal)  geometry.  These 
parameters  are  used  to  generate  data.  They  are  also  used  as  the  starting  guess  for  computing 
reference  fits.  A test  description  window  also  allows  you  to  select  and  define  a form  error,  a 
sampling  pattern,  and  a measurement  error  for  the  given  geometry. 

You  generate  a data  set  by  selecting  the  DATASETInew  command.  At  least  one  test  description 
window  must  be  opened.  The  data  set  is  associated  with  the  last  test  description  window  you 
used.  A data  set  window  is  displayed.  This  window  allows  you  to  select  the  generate  button. 
This  selection  will  cause  the  ATS  to  generate  data  corresponding  to  the  associated  test 
description.  After  the  data  set  is  generated,  the  coordinates  are  displayed  in  the  window. 

You  calculate  a fit  by  selecting  the  fittingInew  menu  command.  A fit  description  window  is 
displayed.  This  window  allows  you  to  select  the  fitting  method  to  be  used  (currently  only  least 
squares  is  implemented).  You  select  the  COMPUTE  FIT  button  to  calculate  the  fit.  You  can  set 
iteration  limits  and  a convergence  factor  to  control  the  fitting  algorithm. 

You  analyze  fit  results  by  selecting  the  analysisInew  command.  An  analysis  window  is 
displayed.  This  window  is  divided  into  three  sections.  In  the  upper  left  is  a box  listing  the  names 
of  all  fit  result  windows  available  for  analysis.  This  list  contains  the  last  fit  window  that  you 
used,  together  with  all  other  fit  windows  associated  with  the  same  geometry  type.  You  can  select 
any  fit  for  detailed  analysis  by  clicking  on  it  with  the  mouse  or  by  highlighting  it  with  the  arrow 
keys  and  pressing  <ENTER>.  If  there  is  more  than  one  fit,  a comparison  to  a reference  fit  is  also 
done.  You  can  select  which  fit  is  used  as  the  reference  by  pressing  R while  the  fit  is  highlighted. 
The  bottom  half  of  the  window  displays  the  detailed  analysis  of  a single  fit.  The  sum  of  the 
squares,  standard  deviation,  maximum,  and  average  of  the  residuals  are  displayed.  The  partial 
derivatives  of  the  objective  function  with  respect  to  the  fitting  parameters  are  shown.  These 
numbers  indicate  how  close  the  fit  is  to  a minimum.  The  upper  right  section  of  the  analysis 
window  contain  the  comparative  results  of  a fit  to  the  selected  reference  fit. 
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4.  DETAILED  INSTRUCTIONS 

This  section  contains  detailed  instructions  for  operating  the  ATS.  Section  4.1  presents  an  overall 
description  of  the  menu  system.  Section  4.2  describes  each  menu  item  and  the  respective  menu 
commands. 

4.1  The  Menu  System 

The  ATS  starts  up  with  a menu  bar  across  the  top  of  the  screen.  The  menu  items  are  listed 
across  this  menu  bar.  The  commands  of  each  menu  item  are  organized  into  pull-down  menus. 
Each  pull-down  menu  contains  a series  of  commands.  The  menu  items  are  PROGRAM,  test, 
DATASET,  FITTING,  and  ANALYSIS.  Whenever  the  menu  bar  is  active,  one  of  the  menu  items  is 
highlighted  to  indicate  which  item  of  the  menu  bar  is  currently  active.  Figure  1 is  a diagram  of 
how  the  menu  system  would  appear  if  all  of  the  menus  were  pulled  down  at  the  same  time.  (The 
ATS  allows  only  one  menu  at  a time  to  be  pulled  down.) 


ATS 

Program 

Test 

Dataset 

Fitting 

Analysis 

About 

Directory 

Quit 

New 

Open 

Save 

Delete 

New 

Open 

Save 

Delete 

Import 

New 

Open 

Save 

Delete 

Import 

New 

Export 

Figure  1:  Menu  bar  from  Algorithm  Testing  System  with  all  menus  pulled  down 


4.1.1  The  Menu  Bar,  the  Mouse,  and  the  Keyboard 

The  ATS  allows  you  to  access  the  system  via  the  menu  bar.  The  menu  bar  can  be  accessed  with 
function  keys  from  the  keyboard,  with  a mouse,  and  with  keystrokes  from  the  keyboard. 

Here  is  how  to  access  the  menu  bar  with  function  keys: 

1 If  no  menu  item  is  highlighted,  press  the  <F-iO>  function  key.  This  action  activates  the 
menu  bar.  (The  menu  bar  must  be  active  to  select  an  item.) 

2 Use  the  <LEFTlRlGHT>  arrow  keys  to  move  along  the  menu  bar  and  select  an  item. 
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3 Use  the  <ENTER>  or  <DOWN>  arrow  key  once  an  item  is  selected  to  pull  down  the 
corresponding  menu. 

4 Use  the  <UP/DOWN>  arrow  keys  to  move  through  and  select  a command  from  the  menu. 

5 Use  the  <enter>  key  to  execute  the  selected  command. 

Here  is  how  to  access  the  menu  bar  with  the  mouse: 

1 Click  on  the  desired  menu  item  to  display  the  menu.  This  will  automatically  activate  the 
menu  bar. 

2 Click  on  the  desired  command  to  be  executed. 

Here  is  how  to  access  the  menu  bar  with  keystrokes  from  the  keyboard: 

1 If  necessary,  press  the  <F-10>  function  key  to  activate  the  menu  bar. 

2 Press  the  highlighted  letter  from  the  desired  menu  item.  This  action  activates  its 
corresponding  command  menu. 

3 Press  the  highlighted  letter  from  the  desired  menu  command.  This  action  executes  the 
corresponding  command. 


The  following  table  lists  the  function  keys  available  to  you. 


Keys 

Function 

A A 

Sequences  through  window  displays. 
Activates  menu  bar. 

Close  active  window  or  menu  display. 

Table  1:  General  function  keys 


The  following  tables  list  the  menu  items  shortcut  keys  and  the  menu  commands  shortcut  keys. 
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Keys 

Menu  item 

Function 

P 

Program  menu 

Takes  you  to  the  Program  menu 

T 

Test  menu 

Takes  you  to  the  Test  menu 

D 

Dataset  menu 

Takes  you  to  the  Dataset  menu 

F 

Fitting  menu 

Takes  you  to  the  Fitting  menu 

A 

Analysis  menu 

Takes  you  to  the  Analysis  menu 

Table  2:  Menu  items  shortcut  keys 


Keys 

Menu 

command 

Function* 

Menus 

N 

New 

Create  a new  object 

Test,  Dataset,  Fitting,  Analysis 

0 

Open 

Open  a saved  object 

Test,  Dataset,  Fitting 

s 

Save 

Save  an  object 

Test,  Dataset,  Fitting 

D 

Delete 

Delete  a saved  object 

Test,  Dataset,  Fitting 

I 

Import 

Import  an  object 

Fitting,  Dataset 

E 

Export 

Export  an  object 

Dataset 

A 

About 

Show  program  information 

Program 

D 

Directory 

Change  directory 

Program 

Q 

Quit 

Exit  the  program 

Program 

*An  object  is  a test 
description,  dataset,  fit,  or 
analysis. 

Table  3:  Menu  commands  hot  keys 


4.2  Menu  Items  and  Commands 

The  menus  shown  in  Figure  1 are  described  in  detail  in  sections  4.2.1  through  4.2.5. 
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4.2.1  The  PROGRAM  Menu 

The  PROGRAM  menu  contains  the  PROGRAmIabout  command,  the  PROGRAmIdirectory 
command,  and  the  PROGRAmIquit  command.  The  PROGRAMIabout  command  displays  some 
general  information  about  the  Algorithm  Testing  System.  It  is  an  information  screen  that  tells 
where,  when  and  by  whom  the  system  was  developed.  The  PROGRAmIdirectory  command 
allows  you  to  change  directories.  For  example,  the  ATS  could  be  running  in  one  directory  and 
the  files  containing  data  sets  could  be  in  another  directory.  The  PROGRAMIQUIT  command  allows 
you  to  exit  the  ATS. 


4.2.2  The  TEST  Menu 

The  TEST  menu  contains  the  commands  for  creating,  opening,  saving,  and  deleting  test 
descriptions.  The  four  available  commands  are  testInew,  testIOPEN,  testIsave,  and 
testIdelete.  These  commands  arc  described  in  detail  in  sections  4.2.2. 1 through  4.2.2.4. 


4.2.2.1  The  test|new  Command 

The  TESTiNEW  command  allows  you  to  create  a new  test  description.  The  information  that  is 
required  to  create  a test  description  is  displayed  in  a window.  The  window  requests  that  a 
geometry  be  defined,  form  errors  identified,  sampling  pattern  chosen,  and  measurement  error  be 
identified.  You  cannot  define  a form  error  or  a sampling  pattern  until  you  have  defined  a 
geometry.  The  information  needed  to  create  a test  description  is  described  in  detail  below.  See 
Appendix  C for  more  detailed  information. 

GEOMETRY  The  GEOMETRY  field  allows  you  to  choose  the  geometry  for 

the  test  description.  Select  geometry  to  obtain  the 
following  options:  LINE,  CIRCLE,  PLANE,  SPHERE,  CYLINDER, 
and  CONE.  When  the  geometry  is  chosen  another  window 
is  displayed.  The  new  window  allows  you  to  enter  a 
location,  a direction,  a size,  an  angle  and  an  extent  for  the 
geometry  (only  those  that  are  relevant  to  the  geometry  are 
displayed).  The  geometry  types  are  described  in  detail  in 
Appendix  C of  this  document. 

FORM  ERROR  The  FORM  ERROR  field  allows  you  to  choose  the  type  of 

form  error  to  be  simulated  in  the  sampling  of  points.  Select 
FORM  ERROR  to  obtain  the  options  for  the  types  of  form 
errors  available.  The  form  errors  vary  depending  on  the 
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type  of  geometry  selected.  The  form  errors  are  described 
in  detail  in  Appendix  C of  this  document. 

SAMPLING  The  SAMPLING  command  allows  you  to  indicate  the 

sampling  pattern  to  be  used  in  generating  data  sets.  Select 
the  SAMPLING  command  to  obtain  the  following  options  for 
sampling  patterns;  RANDOM,  equispaced  with  ends, 
EQUISPACED  NO  ENDS,  and  STRATIFIED.  The  <ENTER>  key 
here  selects  the  next  information  field  to  be  entered  (a 
command  is  not  executed).  The  sampling  patterns  are 
described  in  detail  in  Appendix  C of  this  document. 

MEASUREMENT  ERROR  The  MEASUREMENT  ERROR  field  allows  you  to  indicate  the 

type  of  measurement  error  that  should  be  simulated  to 
generate  the  data  points.  Select  the  MEASUREMENT  ERROR 
command  to  obtain  the  following  options:  NONE,  UNIFORM, 
and  NORMAL.  The  measurement  errors  are  described  in 
detail  in  Appendix  C of  this  document. 


4.2.2.2  The  TEST|OPEN  Command 

The  TESTlOPEN  command  opens  a previously  saved  test  description  (see  TESTISave  below).  Once 
the  TESTlOPEN  command  is  selected,  a new  window  is  displayed.  The  window  shows  a list  of 
names  of  saved  test  descriptions.  You  select  a name  and  a test  description  window  is  displayed 
on  the  screen.  The  test  description  window  will  contain  all  the  information  that  was  stored. 
Press  <ESC>  to  cancel  the  operation. 


4.2^.3  The  test|save  Command 

The  SAVE  command  saves  the  current  test  description.  The  description  is  saved  in  a file  in  the 
current  directory.  (ATS  system  file  names  are  discussed  in  Appendix  B of  this  document)  Once 
a test  description  is  created  you  can  save  it  by  selecting  the  testIsave  command  from  the  TEST 
menu  on  the  main  menu  bar.  You  will  be  prompted  for  a test  description  name.  This  name 
should  be  no  longer  than  eight  characters  and  should  not  be  given  an  extension. 


4.22.4  The  test|delete  Command 

The  TESTiDELETE  command  deletes  a test  description  that  has  been  previously  saved.  Select  the 
TESTIdelete  command  and  then  select  the  test  description  to  be  deleted  from  the  displayed  list. 
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You  must  then  confirm  or  cancel  the  request  by  selecting  OKAY  or  cancel.  Deleting  a test 
description  will  invalidate  any  saved  dataset  that  was  created  from  the  test.  It  is  valid  to  delete 
an  already  open  test  description.  However,  the  test  description  will  be  lost  if  the  window  is 
closed  without  saving  it  again. 


4.2.3  The  DATASET  Menu 

The  DATASET  menu  contains  the  commands  for  creating,  saving,  and  deleting  data  sets.  The  data 
can  be  generated  by  the  data  generator  or  imported  from  an  external  file.  Data  sets  can  also  be 
exported  to  a data  file  (for  use  by  other  programs).  A data  set  is  always  associated  with  and 
dependent  upon  the  test  description.  The  six  available  commands  in  this  menu  are: 
DATASETiNEW,  DATASETlOPEN,  DATASETlSAVE,  DATASETIDELETE,  DATASETI  IMPORT,  and 
DATASETlEXPORT.  These  commands  are  described  in  detail  in  sections  4.2.3. 1 through  4.2.3. 6. 


4.2.3.1  The  dataset|new  Command 

The  DATASETiNEW  command  allows  you  to  generate  a new  data  set.  The  data  set  is  automatically 
associated  with  the  last  active  test  description.  A data  set  cannot  be  generated  without  an  active 
test  description.  A window  is  displayed  when  the  NEW  command  is  selected.  This  window 
allows  you  to  select  the  generate  button.  This  button  invokes  the  data  generator,  which 
generates  data  points  based  on  the  associated  test  description.  After  the  points  are  generated  they 
are  displayed  in  the  window  as  x,  y,  and  z coordinate  values.  You  can  use  the  arrow  keys  or 
page  up  and  down  to  scroll  through  the  points.  (You  must  be  at  the  generate  button  to  be  able 
to  leave  the  window  using  the  keyboard).  If  you  make  changes  to  the  test  description  after  a data 
set  has  been  generated,  selecting  generate  again  will  cause  the  data  set  to  be  replaced  with  new 
data  corresponding  to  the  test  description  values. 


4.2.3.2  The  dataset|OPEN  Command 

The  DATASETlOPEN  command  allows  you  to  open  a previously  saved  data  set  This  command 
also  opens  a window  for  the  test  description  associated  with  the  data  set.  Select  the  open 
command  to  obtain  a list  of  available  data  sets.  Once  a data  set  is  selected,  two  windows  are 
displayed.  One  window  contains  the  test  description  for  the  data  set.  The  other  window  contains 
the  data  set  itself.  (If  the  test  description  is  already  displayed  in  a window,  only  one  new 
window  is  opened.) 
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4.2.3.3  The  dataset|save  Command 

The  DATASETlSAVE  command  saves  the  current  data  set.  Once  the  datasetIsave  command  is 
selected,  a form  is  displayed  for  you  to  enter  the  name  of  the  data  set  to  be  saved.  You  have  the 
option  to  confirm  the  save  by  selecting  okay  or  to  cancel  by  selecting  CANCEL.  If  OKAY  is 
selected  the  data  set  will  be  saved.  The  data  set  name  must  not  exceed  eight  characters  in  length 
and  should  not  be  given  an  extension.  (A  detailed  description  of  the  ATS  system  filenames  is 
discussed  in  Appendix  B of  this  document.) 

4.2.3.4  The  DATASET  [DELETE  Command 

The  DATASETiDELETE  command  deletes  a previously  saved  data  set.  Once  the  delete  command 
is  selected,  a window  is  displayed  with  a list  of  saved  data  sets.  This  window  allows  you  to 
select  the  data  set  to  be  deleted.  You  can  confirm  by  selecting  OKAY  or  cancel  by  selecting 
CANCEL.  The  DATASETiDELETE  command  only  deletes  the  dataset,  not  the  associated  test 
description.  It  is  valid  to  delete  an  already  open  data  set.  However,  the  data  set  will  be  lost  if 
the  window  is  closed  without  saving  it  again. 


4.2.3.5  The  dataset|import  Command 

The  DATASETIimport  command  imports  an  external  file  containing  x-y-z  coordinate  data.  This 
command  displays  a data  set  (data  sets  generated  outside  of  the  ATS)  in  a data  set  description 
window.  In  the  ATS  three  types  of  file  formats  can  be  imported.  The  PTB  A Series,  PTB  B 
Series,  and  the  Generic  format  (The  files  with  PTB  data  sets  are  data  points  that  were 
generated  during  previous  tests  done  in  Germany  by  the  Physikalisch  Technische  Bundesanstalt.) 
Refer  to  Appendix  A for  more  details  on  the  ATS  file  formats. 

The  ATS  allows  you  to  import  these  files  and  run  the  reference  algorithm  on  the  data  sets.  The 
ATS  allows  you  to  run  the  reference  algorithm  with  data  sets  from  different  sources  that 
represent  the  same  geometry.  Once  the  DATASETIimport  command  is  selected,  the  ATS  displays 
a window  that  lists  the  data  set  formats  available.  Generic,  PTB  A Series,  and  PTB  B Series. 
After  selecting  a format  you  enter  the  name  of  the  file  to  be  imported.  If  you  use  wildcard 
characters  (’*’  or  ’?’)  in  the  file  name  a file  selection  window  is  displayed.  You  can  then  select 
the  specific  file  to  be  imported.  Like  the  datasetInew  command,  the  DATASETIimport 
command  will  only  work  if  you  have  already  opened  a test  description  window. 


4.2.3.6  The  dataset|export  Command 

The  DATASETlEXPORT  command  exports  a data  set  to  a file.  The  window  displayed  when 
DATASETIexport  is  selected  allows  you  to  give  the  file  a name.  You  can  enter  a fully  qualified 
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file  name,  with  a path  and  extension  if  desired.  Confirm  the  selection  with  the  okay  button  or 
cancel  with  the  CANCEL  button.  The  DATASETlEXPORT  command  does  not  create  a test 
description  file  and  does  not  add  an  extension  to  the  file  name.  The  datasetIexport  command 
saves  the  file  in  the  Generic  format. 


4.2.4  The  FITTING  Menu 

The  FITTING  menu  contains  the  commands  for  creating,  saving,  and  deleting  fits.  A fit 
description  is  always  associated  with  and  is  dependent  upon  a data  set.  (The  data  set  in  turn  is 
associated  with  a test  description.)  The  five  available  commands  in  this  menu  are:  FITTINgInew, 
FITTINGIOPEN,  FlTTlNGlSAVE,  FlTTlNGlDELETE,  and  FITTINGIIMPORT.  These  commands  are 
described  in  detail  in  sections  4.2.4. 1 through  4.2.4.5. 


4.2.4.1  The  fittingInew  Command 

The  fittingInew  command  allows  you  to  calculate  a fit  using  the  algorithms  that  come  with  the 
ATS.  A fit  cannot  be  calculated  without  an  active  data  set.  The  ATS  currently  implements  the 
least  squares  method  of  fitting.  Each  fit  uses  the  test  description  and  the  data  set  description  to 
obtain  the  data  necessary  to  calculate  the  fit.  The  OBJECTIVE  button  allows  you  to  select  the 
fitting  method  to  be  used  Geast  squares  for  now).  You  can  also  set  an  iteration  limit  and  a 
convergence  factor  for  the  fit.  The  convergence  factor  and  the  iteration  limit  are  discussed  in 
detail  in  the  draft  report,  Least-Squares  Fining  Algorithms  of  the  NIST  Algorithm  Testing  System. 
A positive  convergence  factor  is  an  absolute  convergence  test  A negative  convergence  factor 
is  a relative  convergence  test.  The  COMPUTE  fit  button  is  selected  to  calculate  the  fit.  The 
results  of  the  fit  are  displayed  in  the  fit  description  window.  The  fit  reports  the  sum  of  the 
squares  of  errors  (sse).  The  location  is  reported  in  x,  y,  and  z coordinates.  The  direction,  size 
and  angle  are  reported  for  some  of  the  geometries.  While  the  ATS  is  computing  a fit,  it  displays 
a status  message.  Some  fitting  problems  may  take  considerable  time.  If  you  wish  to  interrupt 
a fit,  press  the  <break>  key  while  holding  down  the  <CTRL>  key.  (On  some  computer  systems, 
there  may  be  a delay  before  this  keystroke  is  recognized  and  the  fit  is  interrupted.) 


4.2.4.2  The  FlTTlNGlOPEN  Command 

The  FlTTlNGlOPEN  command  allows  you  to  open  a file  containing  a previously  saved  fit.  This 
command  causes  the  ATS  to  also  open  the  test  description  window  and  the  data  set  window 
associated  with  the  fit.  Once  the  FlTTlNGlOPEN  command  is  selected,  a window  displays  a list 
of  previously  saved  files.  You  select  a file  and  confirm  or  cancel  using  the  okay  or  cancel 
commands. 
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4.2.4.3  The  fittingIsave  Command 

The  fittingIsave  command  allows  you  to  save  a fit  result  This  command  automatically  saves 
the  test  description  and  the  data  set  associated  to  the  fit  When  the  fittingIsave  command  is 
selected,  a window  is  displayed  in  which  you  enter  a name  for  the  fit.  This  name  should  be  no 
longer  than  eight  characters  and  should  not  be  given  an  extension.  (Refer  to  Appendix  A of  this 
document  for  details  on  the  ATS  file  names.) 


4.2.4.4  The  fittingIdelete  Command 

The  fittingIdelete  command  allows  you  to  delete  a previously  saved  fit  description.  When  the 
fittingIdelete  command  is  selected,  the  ATS  displays  a list  of  saved  fit  descriptions.  This 
window  allows  you  to  select  the  desired  fit  description.  You  then  select  okay  or  cancel  to 
confirm  or  cancel  the  fittingIdelete  request.  Press  <esc>  to  cancel  from  the  list  of  fit 
descriptions. 


4.2.4.5  The  fittingIimport  Command 

The  fittingIimport  command  allows  you  to  import  a fit  description  file.  As  with  fittingInew, 
a data  set  must  be  defined  before  a fit  can  be  imported.  When  the  FITTINGIIMPORT  command  is 
selected,  a window  is  displayed  in  which  you  enter  the  name  of  the  fit  result  file  to  be  imported. 
You  then  confirm  or  cancel  the  FITTINGIIMPORT  request  by  selecting  OKAY  or  cancel.  The  fit 
description  files  that  are  imported  are  fit  results  from  other  sources  (fits  results  that  were 
generated  with  algorithms  other  than  the  reference  algorithms  of  the  ATS).  Refer  to  Appendix 
B of  this  document  for  details  on  the  format  of  the  fit  results. 


4.2  J The  analysis  Menu 

The  analysis  menu  contains  the  command  for  generating  an  analysis  of  the  fit  results.  The  only 
available  command  in  this  menu  is  ANALYSiSiNEW.  This  command  is  described  in  detail  in 
section  4.2.5. 1. 


4.2.5.1  The  analysisInew  Command 

The  analysisInew  command  allows  you  to  generate  an  analysis  of  fit  results.  When  the 
ANALYSiSiNEW  command  is  selected  an  analysis  window  is  displayed.  This  window  is  divided 
into  three  sections. 
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The  upper  left  is  a box  listing  the  names  of  all  fit  result  windows  available  for  analysis.  This 
list  contains  the  last  fit  window  you  used  together  with  all  other  fit  windows  associated  with  the 
same  geometry  type.  You  can  select  any  fit  for  detailed  analysis  by  clicking  on  it  with  the 
mouse  or  by  highlighting  it  with  the  arrow  keys  and  pressing  <ENTER>.  If  there  is  more  than 
one  fit,  a comparison  to  a reference  is  also  done.  You  can  select  the  reference  fit  by  pressing 
R while  the  fit  is  highlighted. 

The  lower  half  of  the  window  displays  the  results  of  a single  fit  analysis.  The  analysis  displays 
the  sum  of  the  squares  of  the  errors  (sse),  the  standard  deviation,  the  maximum  and  average 
residuals,  and  the  gradients.  The  gradients  are  the  partial  derivatives  with  respect  to  x,  y,  and 
z for  the  location,  and  with  respect  to  the  size  and  angles  where  applicable.  The  angles  describe 
the  direction.  The  gradients  will  be  very  small  if  the  fit  is  close  to  a (local)  minimum. 

The  upper  right  section  of  the  window  displays  comparative  analysis  of  two  fit  results.  The 
upper  left  section  of  the  window  lists  the  two  fits.  The  imported  fit  is  listed  first.  The  reference 
fit  is  listed  second  (this  is  the  default).  The  comparative  analysis  displays  the  geometry 
differences  (axis  separation  and  angle  separation),  a residual  analysis  (sse  difference),  and  the 
gradient  difference  (angle  and  length). 

The  analysis  window  displays  a different  single  analysis  result  depending  on  which  fit  is  selected. 
If  an  imported  fit  in  the  upper  left  section  of  the  window  is  selected  then  the  single  analysis 
displayed  is  of  the  imported  fit.  The  comparison  is  of  the  imported  fit  compared  to  the  reference 
fit.  If  the  reference  fit  in  the  upper  left  section  of  the  window  is  selected  then  the  single  analysis 
displayed  is  of  the  reference  fit.  You  can  make  an  imported  fit  the  reference  fit  for  the 
comparison  analysis  by  pressing  R on  the  selected  fit. 


5.  Summary 

The  NIST  Algorithm  Testing  SystenVVersion  1.1  has  been  distributed  for  beta  testing  to  a 
number  of  potential  users.  These  include  members  of  the  ASME  B 89.4. 10  Working  Group  for 
CMS  Software  Performance  Evaluation  and  other  members  of  the  dimensional  metrology 
community.  Users  have  responded  favorably  and  have  contributed  useful  comments  and 
suggestions  which  have  been  considered  during  the  development  of  the  production  ATS.  One 
of  the  recommendations  is  the  need  for  algorithms  other  than  least  squares  in  the  ATS.  NIST 
is  developing  additional  algorithms  for  other  fitting  objectives.  NIST  is  also  developing  a 
procedure  for  submitting  and  selecting  reference  algorithms.  These  algorithms  will  be  the 
performance  baseline  of  the  ATS. 

Any  problems,  questions,  suggestions  or  comments  regarding  the  NIST  Algorithm  Testing  System 
should  be  addressed  to  one  of  the  following  two  people. 
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Cathleen  Diaz 

National  Institute  of  Standards  and  Technology 
Building  220  Room  A 127 
Gaithersburg,  MD  20899 

email  diaz@cme.nist.gov 
Telephone  (301)  975  - 2889 
Fax  (301)  258  - 9749 

Theodore  H.  Hopp 

National  Institute  of  Standards  and  Technology 
Building  220  Room  A 127 
Gaithersburg,  MD  20899 

email  hopp@cme.nist.gov 
Telephone  (301)  975  - 3545 
Fax  (301)  258  - 9749 
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APPENDIX  A:  ATS  FILE  FORMATS 


A.1  INTRODUCTION 

The  Algorithm  Testing  System  provides  several  mechanisms  for  bringing  data  into  and  sending 
data  out  of  the  system.  The  DATASETIimport  command  enables  you  to  access  data  sets  generated 
outside  the  ATS,  while  the  datasetIexport  command  enables  you  to  save  data  sets  generated 
by  the  ATS.  This  export  file  may  be  used  as  input  to  your  algorithm.  To  bring  the  fit  results 
generated  by  your  algorithm  into  the  ATS,  use  the  FITTINGIimport  command. 

This  appendix  describes  the  file  format  requirements  for  importing  and  exporting  data  sets,  as 
well  as  importing  fit  results. 


A.2  DATASET  IMPORT  FILE  FORMATS 

Three  types  of  data  sets  may  be  imported  into  the  ATS.  The  formats  supported  are  PTB  A 
Series,  PTB  B Series,  and  Generic  format.  Both  PTB  A and  B Series  data  sets  are  included  in 
the  ATS  distribution.  The  Generic  format  allows  you  to  import  data  from  a measurement  system 
or  other  source.  The  following  three  sections  describe  the  import  formats  in  detail. 


A.2.1  PTB  A Series 

The  files  containing  data  sets  in  the  PTB  A Series  have  the  following  format: 

<geometry  type>  <data  set  number>  <time  stamp> 
xl -value  yl -value  zl -value 
x2-value  y2-value  z2-value 
x3-value  y3-value  z3-value 


The  first  line  of  the  file  contains  the  geometry  type,  the  data  set  number,  and  a time  stamp.  The 
following  lines  list  the  coordinate  values  of  the  data  set.  An  example  follows: 

LINE  1 881005 

62.0007  3.5427  12.1164 

69.4374  3.5504  12.1164 

76.8742  3.5574  12.1164 
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Each  data  set  from  the  PTB  A Series  is  in  a separate  file.  The  file  name  convention  is 
DATASET.A#  where  # is  the  data  set  number  and  A means  it  is  from  the  PTB  A Series.  For 
instance,  the  file  name  for  the  previous  example  is  DATASET.Al. 


A.2.2  PTB  B Series 

The  distribution  of  the  PTB  B Series  data  includes  two  files  for  each  data  set.  One  file  contains 
the  data  set,  while  a comment  file  contains  detailed  information  about  the  data  set.  The  file 
containing  the  data  set  has  the  following  format: 

xl -value  yl -value  zl -value 
x2-value  y2-value  z2-value 
x3-value  y3-value  z3-value 


An  example  of  a file  containing  a PTB  B Series  data  set  follows: 

746.0352  167.2775  -81.9667 
798.8503  135.0613  -127.6552 
800.1427  123.9985  -163.9518 
821.0459  168.8838  -107.2863 


The  corresponding  comment  file  indicates  that  the  data  set  contains  points  on  a cylinder.  The 
comment  file  also  contains  details  on  the  approximate  dimensions  and  position  of  the  geometric 
element.  Each  data  set  from  the  PTB  B Series  is  in  a separate  file.  Similarly,  the  detailed 
information  for  each  data  set  is  in  a separate  comment  file.  The  files  containing  data  set  are 
named  DATASET.B#  where  # is  the  data  set  number  and  B means  it  is  from  the  PTB  B Series. 
The  files  containing  detailed  information  for  a data  set  are  name  COMMENT.B#  where  # is 
number  for  the  PTB  B Series  data  set.  For  instance,  the  file  name  for  the  previous  example  is 
DATASET.B20,  and  its  corresponding  comment  file  is  COMMENT.B20. 


A.2.3  Generic  Format 

The  files  containing  data  sets  in  the  Generic  fonnat  have  the  following  structure: 


A2 


APPENDIX  A:  ATS  FILE  FORMATS 


<number  of  points> 
xl -value  yl -value 
x2-value  y2-value 
x3-value  z3-value 


zl -value 
z2-value 
z3-value 


The  first  line  of  the  file  lists  the  number  of  points  that  are  contained  in  the  data  set.  The  rest  of 
the  file  are  the  x,  y,  and  z coordinates  of  each  point,  with  one  point  per  line.  An  example  of  a 
data  set  file  in  the  Generic  format  follows: 


5 


23.4756 

3.5678 

-42.9876 

20.6543 

3.5789 

-41.9875 

17.7654 

3.5876 

-40.6532 

14.8743 

3.5902 

-39.8765 

11.6273 

3.6087 

-38.4143 

The  file  names  for  the  Generic  format  can  be  any  valid  DOS  file  name. 

A.3  DATASET  EXPORT  FILE  FORMATS 

Use  the  datasetIexport  command  to  transfer  ATS-generated  data  to  the  system  that  contains 
your  algorithms.  This  export  file  is  in  Generic  format.  See  Section  A.2.3  for  more  details. 


A.4  FITTING  IMPORT  FILE  FORMATS 

Fit  results  from  test  algorithms  are  imported  by  using  files.  Each  file  contains  a single  fit  result 
that  is  comprised  of  an  appropriate  number  of  fit  parameters  according  to  the  type  of  geometry. 
These  parameters  vary  among  the  different  geometry  types.  Table  A.l  specifies  the  parameters 
to  be  reported  for  each  geometry.  Within  the  import  file  for  fit  results,  the  parameter  values  are 
separated  by  white  space  (such  as  spaces,  tabs,  or  end-of-line  characters).  Any  other  separation 
characters  (such  as  commas)  will  cause  unpredictable  results.  These  parameter  values  may  be 
expressed  by  decimal  floating  point  or  exponential  notation.  The  ATS  ignores  information  or 
comments  in  the  file  beyond  the  last  parameter. 
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GEOMETRY 

FILE  FORMAT 

PARAMETER 

NUMBER 

PARAMETER 

NAME 

PARAMETER 

DESCRIPTION 

LINE 

1.2.3 

<a:>  <>’>  <z> 

coordinates  of  a point  on  line 

4,  5.6 

<a>  <b>  <c> 

direction  numbers  of  line 

CIRCLE 

1,  2,3 

<x>  <y>  <z> 

coordinates  of  center 

4.  5,6 

<a>  <b>  <c> 

direction  numbers  of  the  normal 

7 

<d> 

diameter 

PLANE 

1,  2,3 

<x>  <y>  <z> 

coordinates  of  a point  on  plane 

4,  5.6 

<a>  <b>  <c> 

direction  numbers  of  plane  normal 

SPHERE 

1,2,3 

<x>  <y>  <z> 

coordinates  of  center 

4 

<d> 

diameter 

CYLINDER 

1,2,3 

<x>  <y>  <z> 

coordinates  of  a point  on  axis 

4,  5,6 

<a>  <b>  <c> 

direction  numbers  of  axis 

7 

<d> 

diameter 

CONE 

1.2.3 

<x>  <y>  <z> 

coordinates  of  a point  on  axis 

4,  5.6 

<a>  <b>  <c> 

direction  numbers  of  axis 

7 

<d> 

diameter  at  perpendicular  cross 
section  through  point  specified  by 
parameters  1-3. 

8 

<\|/> 

full  cone  angle 
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B.l  INTRODUCTION 

The  Algorithm  Testing  System  accesses  data  files  when  saving,  deleting,  importing,  and  exporting 
data.  There  are  test  description  files,  data  set  description  files,  and  fit  description  files.  You  can 
give  each  of  these  files  a file  name  or  you  can  let  the  ATS  give  each  one  a default  file  name. 
Each  window  that  is  displayed  in  the  ATS  has  a default  name  from  which  the  default  file  name 
is  generated.  This  appendix  describes  the  different  file  names  in  the  ATS. 


B.2  DEFAULT  FILE  NAMES 

A default  name  is  given  to  any  new  window  displayed  in  the  ATS.  Following  is  a list  of  these 
types  of  file  names: 

(TSTOOO) 

(DSDOOO) 

(FITOOO) 

(ANAOOO) 

(TSTOOl) 

The  test  description  window  is  denoted  with  (TST###).  The  window  that  allows  you  to  generate 
the  data  set  for  the  test  description  is  denoted  (DSD###).  The  fitting  window  is  denoted  with 
(FIT###)  and  the  analysis  window  is  denoted  with  (ANA###).  The  ###  symbol  is  a number  in 
each  of  the  file  names.  All  numbers  are  used  to  distinguish  open  windows  of  the  same  type. 
Windows  corresponding  to  saved  objects  have  as  window  names,  the  name  of  the  saved  object. 

If  a test  description,  a data  set,  or  a fit  result  is  saved  and  you  do  not  provide  the  ATS  with  a 
file  name,  the  ATS  will  generate  a default  file  name.  Following  is  an  example  of  the  default  file 
names: 


(TSTOOO).TSD 

(DSD000).DSD 

(FIT000).FD 

(TSTOOl).TSD 


You  can  choose  any  valid  DOS  name  for  the  file  being  saved.  The  ATS  will  add  the  appropriate 
extension.  In  other  words,  the  ATS  will  always  automatically  add  an  extension  to  the  file  name. 
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The  following  table  summarizes  default  file  names. 


display  name 

default  saved  name 

saved  name 

test  descriptions 

(TST###) 

(TST###).TSD 

NAME.TSD 

data  sets 

(DSD###) 

(DSD###).DSD 

NAME.DSD 

fit  results 

(FIT###) 

(Frr###).FD 

NAME.FD 

analysis 

(ANA###) 

N/A 

N/A 

TABLE  1:  Default  File  Names 


B3  DATA  SETS 

Data  sets  are  imponed  and  exported  in  three  file  formats:  PTB  A,  PTB  B,  and  Generic.  In  this 
distribution  of  the  ATS  we  have  included  some  files  with  data  sets  in  the  PTB  A Series  and  the 
PTB  B Series.  The  file  names  in  the  PTB  A Series  are  denoted  DATASET.A#.  The  name 
itself  tells  you  that  it  is  a data  set.  The  A in  the  extension  means  it  is  from  the  PTB  A Series 
and  the  number  is  a unique  identifier.  The  file  names  in  the  PTB  B Series  are  denoted 
DATASET.B#.  The  name  itself  tells  you  that  it  is  a data  set.  The  B in  the  extension  means  it 
is  from  the  PTB  B Series  and  the  number  is  a unique  identifier.  The  files  in  the  PTB  B Series 
have  associated  comment  files.  These  comment  files  are  denoted  COMMENT.B#.  Following 
is  a list  with  examples  of  these  types  of  file  names. 

DATASET.A12 

DATASET.B1 

COMMENT.B1 

DATASET.B30 

COMMENT.B30 

DATASET.A5 

There  are  also  Generic  data  set  files.  These  file  names  are  denoted  with  any  valid  DOS  name. 
For  example: 

TESTDATA.DAT 

NEWDATA 

are  file  names  for  data  sets  of  the  Generic  format.  In  the  Generic  format,  one  line  has  the 
number  of  points  in  the  file  and  the  rest  of  the  file  contains  the  points. 
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B.4  OTHER  FILE  NAMES 

The  ATS  executable  is  denoted  ATS. EXE.  The  •.DAT  file  names  denote  files  with  data  sets  in 
the  Generic  format.  The  Generic  format  is  the  default  file  type  when  a data  set  is  exponed. 
The  file  contains  a line  with  the  number  of  points  in  the  data  set  and  the  data  set  itself.  Each 
value  is  separated  by  a white  space.  Files  in  the  Generic  format  can  also  be  imported. 
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Algorithm  Testing  System  Data  Generator 

Geometry  Types,  Form  Errors,  Sampling  Patterns, 
and  Measurement  Errors 
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C.l  INTRODUCTION 

The  Algorithm  Testing  System  allows  you  to  generate  data  for  your  tests  using  the  Data 
Generator.  This  Appendix  describes  the  Data  Generator  in  detail.  The  Data  Generator  allows 
you  to  generate  data  based  on  geometry  types,  form  errors,  sampling  patterns,  and  measurement 
errors.  The  following  sections  describe  the  specifics  of  the  Data  Generator. 


C.2  GEOMETRY  TYPES 

The  ATS  Data  Generator  supports  the  following  geometry  types:  lines,  circles,  planes,  spheres, 
cylinders,  and  cones.  Each  geometry  is  defined  by  a set  of  parameters  which  describes,  when 
applicable,  the  geometry’s  location,  direction,  size,  angle,  and  extent.  Table  C.l  lists  the 
parameters  necessary  for  describing  each  geometry  type.  Note  that  the  size  parameters  are  radii, 
not  diameters,  and  the  angle  of  a cone  is  the  half-angle.  These  parameters  are  different  from  the 
corresponding  parameters  in  the  impon  format  for  fits,  where  diameters  and  full  cone  angle  are 
expected.  (See  Appendix  A.) 

Geometry  definitions  are  specified  relative  to  a world  coordinate  system.  Each  geometry  has  a 
local  coordinate  system  determined  by  the  geometry  definition.  The  local  coordinate  system  for 
each  geometry  type  is  defined  relative  to  the  world  coordinate  system.  This  is  done  by  defining 
a local  X and  a local  z axis  as  shown  in  the  following  table. 


geometry  type 

local  X 

local  z 

line 

line  direction 

projected  world  z (or  x)^ 

circle 

projected  world  x (or  y)^ 

normal  to  plane  of  circle 

plane 

projected  world  x (or  y)^ 

normal  to  plane 

sphere 

world  X 

world  z 

cylinder 

projected  world  x (or  yY 

axis  direction 

cone 

projected  world  x (or  y)^ 

axis  direction 

^The  world  axis  is  projected  into  a plane  perpendicular  to  the  other  local  axis  to  obtain  the 
indicated  local  axis.  The  alternate  world  axis  is  used  when  the  other  local  axis  is  close  to  the 
default  world  axis.  ("Qose  to"  means  that  the  dot  product  has  magnitude  at  least  0.9.) 

The  local  y axis  is  defined  so  that  the  local  coordinate  system  is  right-handed. 
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C.3  FORM  ERRORS 

The  ATS  can  simulate  any  one  of  a number  of  form  errors.  The  types  of  form  errors  available 
vary  among  the  different  geometry  types.  The  types  of  form  errors,  and  the  geometry  types  to 
which  they  apply,  are  shown  in  Table  C.2.  The  ATS  will  not  allow  you  to  define  a form  error 
until  a geometry  type  has  been  specified.  Detailed  information  about  the  parameters  that  define 
each  form  error  type  can  be  found  in  the  report  Form  Error  Models  of  the  NIST  Algorithm 
Testing  System}  TTiat  report  is  included  in  the  ATS  distribution. 


C.4  SAMPLING  PATTERNS 

The  ATS  supports  four  types  of  sampling:  random,  equispaced  with  end  points,  equispaced 
without  end  points,  and  stratified.  All  four  sampling  patterns  are  supported  for  all  geometries, 
although  they  mean  slightly  different  things  for  each  geometry  type.  The  ATS  will  not  allow  you 
to  define  a sampling  pattern  until  a geometry  type  has  been  specified. 

In  the  description  below,  geometry  extents  are  based  on  Table  C.l.  In  particular,  for  circles, 
spheres,  cylinders,  and  cones,  if  u^in=Un,„,  is  treated  as  having  the  value  Uniin+360®. 

Random  sampling  means  that  each  data  point  is  selected  at  random  from  within  the  extent  of  the 
geometry.  For  circles  and  lines,  a uniformly-distributed  random  number  is  generated  between 
u^ia  and  Un»„.  For  surfaces,  two  random  numbers  are  generated,  one  for  u and  one  for  v. 

Equispaced  sampling  can  be  done  with  or  without  endpoints.  In  both  cases,  the  surface  is 
divided  into  a grid  by  dividing  the  extent  into  bins  according  to  the  number  of  points  indicated 
in  u (and  v for  surfaces).  Equispaced  sampling  with  end  points  means  that  a sample  point  is 
generated  at  each  grid  intersection  point  ^uispaced  sampling  without  end  points  means  that 
a sample  point  is  generated  in  the  center  (in  u-v  space)  of  each  box  in  the  grid. 

Stratified  sampling  is  similar  to  equispaced  sampling  without  end  points.  Rather  than  sampling 
at  the  center  of  each  box,  however,  each  sample  point  is  obtained  by  generating  a uniformly- 
distributed  random  number  (or  two  numbers,  for  surfaces)  between  the  limits  of  the  grid  box. 

For  spheres  and  cones,  it  may  happen  that  several  grid  points  in  u-v  space  correspond  to  the  same 
x-y-z  coordinates.  For  instance,  if  vs=-90°  on  a sphere,  all  values  of  u correspond  to  the  south 
pole.  A similar  problem  occurs  at  the  other  pole  of  the  sphere  and  at  the  apex  of  a cone.  This 


*Algeo,  M.  E.  A.  and  Hopp,  T.  H.,  Form  Error  Models  of  the  NIST  Algorithm  Testing 
System,  NISTTR  4740,  National  Institute  of  Standards  and  Technology,  Gaithersburg,  MD, 
January  1992. 
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problem  only  arises  for  spheres  when  using  equispaced  sampling  with  end  points.  It  can  occur 
in  cones  for  either  type  of  equispaced  sampling.  In  all  cases  where  the  sampling  pattern  defines 
multiple  points  at  the  same  x-y-z  coordinates,  the  ATS  will  generate  only  one  point  at  that 
position.  This  means  that  the  actual  number  of  points  generated  in  a data  set  may  be  less  than 
the  number  defined  in  the  sampling  plan. 

C.5  MEASUREMENT  ERROR 

Measurement  error  is  a displacement  of  a sampled  point.  The  ATS  adds  a measurement  error 
to  each  sample  point  after  it  is  generated.  The  measurement  error  at  a point  is  independent  of 
the  error  at  any  other  point,  and  is  also  independent  of  the  geometry.  All  measurement  errors 
arc  random.  You  can  control  whether  measurement  errors  are  to  be  added,  and  you  can  define 
the  distribution  for  the  magnitude  of  the  measurement  error.  Two  distributions  are  supported: 
uniform  and  normal.  Each  error  is  added  in  a random  direction,  with  approximately  uniform 
distribution. 
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Table  C.2  Application  of  form  errors 


LINE 

CIRCLE 

PLANE 

SPHERE 

CYLINDER 

CONE 

SIP^ 

X 

X 

X 

X 

STEP 

X 

X 

X 

BEND 

X 

X 

RANDOM 

X 

X 

X 

X 

X 

X 

AXIAL  SINE 

' 

X 

X 

RADIAL  SINE 

X 

X 

AXIAL  RADIUS 

SINE 

liiiiiiiilili 

X 

X 

AXIAL  STEP 

X 

X 

RADIAL  STEP 

X 

X 

AXIAL  RADIAL 

STEP 

lllllllllll 

X 

X 

C5 


", 


1 


j 

\ I ,; 

y 

^ ..-  J . r ^ 

.,  ,.', ■■?•'-)■  . I 

, . ''it  w'lvj  f ' 

It  1}  ' " , ■ tM\J‘  ;<'  ••‘'/‘^'■'"j^\<*‘'iy  -v,, 

-■  „ ■ \l  ^ 

, ■■■|  L'»- 


'■.?j 


’ I . iTfyi  - _ 

■j^ewr.  ■■■■.■«rrf«*«l*it.): 


■ 


'n'. 


j: 


Appendix  D 
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D.l  INTRODUCTION 

The  Algorithm  Testing  System  allows  you  to  analyze  and  compare  fit  results.  This  appendix 
describes  the  details  displayed  in  the  analysis  window  of  the  ATS. 


D.2  FIT  RESULTS  LIST 

The  upper  left  section  of  the  analysis  window  displays  the  names  of  the  fits  computed.  This  list 
contains  the  name  of  the  last  window  you  used  together  with  the  names  of  all  other  fit  windows 
associated  with  the  same  geometry  type.  Each  fit  can  have  a single  analysis  done,  one  at  a time. 
You  do  this  by  selecting  a fit  with  the  mouse  or  by  highlighting  it.  Any  two  fits  can  be 
compared  and  any  one  of  the  two  can  be  the  reference.  You  select  the  reference  fit  by  pressing 
R while  the  fit  is  highlighted. 

D.3  SINGLE  ANALYSIS 

The  lower  half  of  the  window  displays  the  results  of  a single  fit  analysis.  The  analysis  displays 
tlie  sum  of  the  squares  of  the  errors  (sse),  the  standard  deviation,  the  maximum  and  average 
residuals,  and  the  gradients.  The  gradients  are  the  partial  derivatives  with  respect  to  x,  y,  and 
z for  the  location,  and  with  respect  to  the  size  and  angles  where  applicable.  The  angles  describe 
the  direction.  The  gradients  will  be  very  small  if  the  fit  is  close  to  a (local)  minimum. 

The  following  table  lists  the  parameters  that  describe  the  details  in  the  single  analysis. 


Sum  of  the  Squares  of  Errors  (sse) 

Standard  Deviation 

E'^ 

N 

Maximum  Residual 

Max.\e.\ 

Average  Residual 

1 

TABLE  1:  Single  Analysis  Parameters 
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The  following  table  lists  the  equations  that  describe  the  gradients  in  the  single  analysis.  Note 
that  sse  is  the  sum  of  the  squares  of  errors. 


Gradient  with  respect  to  x 

dsse 

dx 

Gradient  with  respect  to  y 

dsse 

dy 

Gradient  with  respect  to  z 

dsse 

dz 

Gradient  with  respect  to  size 

dsse 

dsize 

Gradient  with  respect  to  the  angle 

dsse 

dangle 

Angle  from  z axis 

dsse 

do. 

Angle  from  z-x  plane 

dsse 

'W 

TABLE  2:  Single  Analysis  Gradient  Parameters 


D.4  COMPARATIVE  ANALYSIS 

The  comparative  analysis  is  presented  by  computing  the  geometry  differences  between  the  test 
fit  and  the  reference  fit.  The  computations  displayed  for  the  geometry  difference  is  dependent 
upon  the  geometry  type. 

The  following  table  lists  the  specific  details  that  are  displayed  for  each  geometry  type.  The  X’s 
in  the  table  indicate  which  details  are  listed  for  each  geometry  type. 
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line 

circle 

plane 

sphere 

cylinder 

cone 

separation 

X 

X 

X 

X 

X 

X 

angle 

X 

X 

X 

X 

X 

size 

X 

:|||||||||:||||:||| 

X 

X 

X 

form 

||||||i|||i:|p|||| 

X 

TABLE  2:  Geometry  Difference  Specifics 


D.4.1  Geometry  Difference 

The  separation  difference  is  the  perpendicular  distance  from  the  point  that  locates  the  test  fit  to 
the  different  points  on  the  reference.  This  location  varies  with  the  geometry  type. 

The  following  table  lists  where  the  point  on  the  reference  lies. 


geometry 

reference 

line 

line 

circle 

center 

plane 

plane 

sphere 

center 

cylinder 

axis 

cone 

axis 

TABLE  3:  Separation  Difference  in  Geometry 

The  angle  difference  is  the  angle  that  is  formed  between  the  two  geometry  fits.  For  example  in 
the  case  of  a cone,  the  angle  is  the  one  formed  between  the  axis  of  the  test  fit  and  the  axis  of 
the  reference  fit.  The  size  difference  is  the  difference  between  the  test  fit  radius  and  the 
reference  fit  radius  (if  applicable  to  the  geometry  type).  The  form  difference  is  the 

difference  in  angles  between  the  two  forms,  the  reference  fit  and  the  test  fit. 
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D.4.2  Residual  Analysis 

The  residual  analysis  in  the  comparative  section  of  the  window  displays  the  sum  of  the  errors 
difference  and  the  gradient  difference  between  the  two  geometries. 

Following  is  a table  containing  the  equations  that  represent  the  comparative  parameters  in  the 
residual  analysis. 


Sum  of  the  Squares  of  Errors  Difference  (sse) 

Gradient  of  the  Test  Fit  (Vj) 

v(E 

Gradient  of  the  Reference  Fit  (Vr) 

v(E 

Gradient  Difference:  angle 

COS"^(  ) 

Gradient  Difference:  length 

TABLE  4:  Residual  Analysis  Equations 

The  sse  difference  is  the  difference  between  the  sse’s  of  each  fit,  the  test  fit  and  the  reference 
fit.  The  gradient  difference  between  the  geometries  is  computed  with  respect  to  the  angle  and 
to  the  length.  The  gradient  of  the  sse  is  computed  for  the  test  fit  and  the  reference  fit.  These 
results  are  used  to  calculate  the  gradient  differences  with  respect  to  the  angle  and  to  the  length. 
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